v|\ United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


10/716,616 


11/20/2003 


Henricus A. Marqucring 


0142-0436P 


6357 



2292 7590 10/18/2007 

BIRCH STEWART KOLASCH & BIRCH 
PO BOX 747 

FALLS CHURCH, VA 22040-0747 



EXAMINER 



DALEY, CLIFTON G 



ART UNIT 



PAPER NUMBER 



2624 



NOTIFICATION DATE 



DELIVERY MODE 



10/18/2007 ELECTRONIC 

Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 

Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the 
following e-mail address(es): 

mailroom@bskb.com 



PTOL-90A (Rev. 04/07) 



Office Action Summary 


Application No. 

10/716,616 


Applicant(s) 

MARQUERINGETAL 


Examiner 

Clifton G. Daley 


Art Unit 

2624 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S. C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 20 November 2003 . 
2a)D This action is FINAL. 2b)S This action is non-final. 

3) Q Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) E3 Claim(s) 1-21 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) . is/are allowed. 

6) M Claim(s) 1-21 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. . 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)^ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
aM All b)D Some * c)D None of: 

1.13 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) Notice of References Cited (PTO-892) 

2) IZI Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) ^ Information Disclosure Statement(s) (PTO/SB/08) 

Paper No(s)/Mail Date 11/20/2003. 5/3K004 . 



4) n Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) □ Notice of Informal Patent Application 

6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 08-06) 



Office Action Summary 



Part of Paper No./Mail Date 20070927 



Application/Control Number: 10/716,616 Page 2 
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DETAILED ACTION 

Claim Rejections - 35 (JSC § 102 

1. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 

form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

2. Claims 1-3, 11, 14, 15 and 18 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Belaid et al. (Hereinafter "Belaid": Y. Belaid, A. Belaid and E. Turolla, 
"Item Searching in Forms: Application to French Tax Form", 1995, IEEE Proceedings of 
the third international conference on Document analysis and recognition, Montreal, 
Que., Canada, pages 744-747 (August 14-16, 1995)). 

Regarding claims 1, 14 and 18, Belaid teaches a method, equivalent 
computer program product embodied on at least one computer readable medium, and 
equivalent device, of segmenting a composite image of pixels into a number of fields 
corresponding to lay-out elements of the image, the pixels having a value representing 
the intensity and/or color of a picture element, the method comprising: constructing 
separating elements corresponding to rectangular areas of adjacent pixels of the image 
having a background property indicative of a background of the image (page 745, Fig. 
2, i.e. real segments of the Hough line, corresponding black lines 
representing background elements of the image); and constructing a graph 
representing the lay-out elements of the image by defining vertices of the graph on the 
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basis of intersections of separating elements that are substantially oriented in 
predetermined separation directions (page 745, § 3, 1 1), defining edges of the graph 
between the vertices corresponding to the separating elements (page 745, § 3, D 1, 
i.e. arc link), and defining field separators corresponding to the edges of the graph 
(page 744, § 2, U 1, i.e. cell boundaries). 

Regarding claims 2 and 15, Belaid teaches the method and equivalent 
computer program product as claimed in claim 1, wherein the step of defining vertices 
comprises: constructing subsets of the separating elements that are substantially 
oriented in the predetermined separation directions (page 745, Fig. 2, i.e. horizontal 
and vertical), and determining the intersections between pairs of separating elements 
from the subsets (page 745, § 3, U 2). 

Regarding claim 3, Belaid teaches the method as claimed in claim 2, wherein 
the step of determining the intersections comprises: determining an area of overlap of 
the separating elements from the subsets (page 745, § 3, U 2, lines 1-3, i.e. the 
intersection of the vertical and horizontal lines), and locating the vertex at the 
center of the area of overlap (i.e. the intersection of the corresponding Hough 
lines which are centered in their respective vertical and horizontal strips, 
page 745, left column, lines 18-22). 

Regarding claim 11, Belaid teaches the method as claimed in claim 1, wherein 
the step of constructing the separating elements comprises: processing the image in 
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two orthogonal separation directions (page 745 § 2, U 2, lines 7-9, i.e. vertical and 
horizontal directions). 



Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented 
and the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Belaid as 
applied to claim 1 above, and further in view of Abdel-Mottaleb et al. (Hereinafter 
"Abdel-Mottaleb": US 6263113). 

Belaid teaches the method as claimed in claim 1. 

Belaid does not teach the limitation wherein the graph constructing step 
comprises: assigning a weight to the edges indicating an Euclidean distance between 
the vertices. 

However Abdel-Mottaleb discloses a graph constructing step comprising: 
assigning a weight to the edges indicating an Euclidean distance between the vertices 
(column 5, lines 31-33). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Abdel-Mottaleb's step to Belaid's graph 
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construction, the motivation being to detect form cells in a complex background 
(Abdel-Mottaleb: column 1, lines 48-50). 

5. Claims 5, 10, 17 and 20 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Belaid as applied to claim 1 above, and further in view of 
Antonacopoulos et al. (Hereinafter "Antonacopoulos": A. Antonacopoulos and R. T. 
Ritchings, "Flexible Page Segmentation Using the Background", 1994, IEEE Proceedings 
of Computer Vision and Image Processing, Vol. 2, pp. 339-344). 

Regarding claims 5, 17 and 20, Belaid teaches the method, equivalent 
computer program product, and equivalent device, as claimed in claim 1. 

Belaid does not teach the limitation to the method as further comprising: 
constructing a set of maximal rectangles, a maximal rectangle being a rectangular part 
of the image in one of the separation directions, that has the maximum possible area 
without including a pixel not having the background property indicative of a background 
of the image; and constructing the separating elements in a cleaning step wherein at 
least one pair of overlapping maximal rectangles in the set is replaced by an informative 
rectangle that is a rectangular part of an area combining the areas of the pair, the 
rectangular part having the maximum possible length in the relevant separation 
direction. 

However Antonacopoulos discloses an image segmentation method comprising 
the steps of constructing a set of maximal rectangles, a maximal rectangle being a 
rectangular part of the image in one of the separation directions, that has the maximum 
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possible area without including a pixel not having the background property indicative of 
a background of the image (page 341, § 2.2, H 1); and constructing the separating 
elements in a cleaning step wherein at least one pair of overlapping maximal rectangles 
in the set is replaced by an informative rectangle that is a rectangular part of an area 
combining the areas of the pair, the rectangular part having the maximum possible 
length in the relevant separation direction (page 341, Figure 4, white tile A). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Antonacopoulos' maximal rectangle steps 
to Belaid's image segmentation method, the motivation being to identify white space 
separators (Belaid: Page 746, § 5, HI, lines 9-11). 

Regarding claim 10, Belaid in combination with Antonacopoulos teaches the 
method as claimed in claim 5, wherein the step of constructing the maximal rectangles 
comprises: determining a list of maximal runs, a maximal run being a straight line of 
pixels having the background property, the line having the maximum possible length 
without including a pixel not having the background property, taking a specific maximal 
run from the list as rectangle, testing the rectangle if extension is possible by 
determining for a next maximal run if the next maximal run comprises pixels adjacent 
to pixels of the rectangle in a width direction, if the extension is possible, extending the 
rectangle by constructing a new rectangle having the maximum area including pixels of 
the rectangle and the next maximal run, if no extension is possible, adding the 
rectangle to the set of maximal rectangles, and eliminating from the list any maximal 
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run that is completely contained in the new rectangle (Antonacopoulos: page 341, § 
2.2 H 2-4, i.e. the Antonacopoulos algorithm is substantially the same, and 
produces the same result as the recited algorithm. While Antonacopoulos 
does not explicitly disclose the use of a list in his algorithm, the use of lists to 
facilitate adding, sorting and removing items was well know in the art at the 
time of the invention and would therefore have been obvious to try). 
6. Claim 6 is rejected under 35 U : S.C. 103(a) as being unpatentable over Belaid in 
combination with Antonacopoulos as applied to claim 5 above, and further in view of 
Katsuyama (US 6226402). 

Belaid in combination with Antonacopoulos teaches the method as claimed in 
claim 5. 

Belaid in combination with Antonacopoulos does not teach the limitation wherein 
the cleaning step further comprises at least one of the following: deleting a maximal 
rectangle having a length below a predefined value, deleting a maximal rectangle 
having a width below a predefined value, and deleting a maximal rectangle having an 
aspect ratio below a predefined value, the aspect ratio being a longer side length 
divided by a shorter side length. 

However Katsuyama discloses a cleaning step further comprising at least one of 
the following: deleting a maximal rectangle having a length below a predefined value, 
deleting a maximal rectangle having a width below a predefined value, and deleting a 
maximal rectangle having an aspect ratio below a predefined value, the aspect ratio 
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being a longer side length divided by a shorter side length (See Fig. 16, for the 
horizontal case, and note that the same process can be applied to the vertical 
case (column 16, lines 4 and 5)). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Katsuyama's deleting method to the 
cleaning step of Belaid combined with Antonacopoulos, the motivation being to extract 
ruled line equivalent separators (Katsuyama: column 2, lines 62-65). 
7. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Belaid in 
combination with Antonacopoulos as applied to claim 5 above, and further in view of 
McLeod et al. (Hereinafter "McLeod": US 5778092). 

Belaid in combination with Antonacopoulos teaches the method as claimed in 
claim 5. 

Belaid in combination with Antonacopoulos does not teach the limitation wherein, 
prior to the step of constructing the maximal rectangles, the image is preprocessed by 
at least one of the following: removing noise by adapting the value of isolated deviant 
pixels to an average value of pixels in the neighborhood, halftoning by transforming the 
pixels to either white or black, and reducing the number of pixels by downsampling. 

However McLeod discloses a preprocessing step wherein the image is 
preprocessed by at least one of the following: removing noise by adapting the value of 
isolated deviant pixels to an average value of pixels in the neighborhood (column 10, 
line 25), halftoning by transforming the pixels to either white or black (column 9, 
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lines 57-60, i.e. thresholding), and reducing the number of pixels by downsampling 
(column 2, lines 6-11, i.e. reduced resolution). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied McLeod's preprocessing method prior to 
the step of constructing the maximal rectangles of Belaid combined with 
Antonacopoulos, the motivation being to enhance separation of text and images from 
page background (McLeod: column 10, lines 43-45). 

8. Claims 8 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Belaid in combination with Antonacopoulos as applied to claim 5 above, and further in 
view of Huang et al. (Hereinafter "Huang": US 5416849). 

Regarding claim 8, Belaid in combination with Antonacopoulos teaches the 
method as claimed in claim 5. 

Belaid in combination with Antonacopoulos does not teach the limitation wherein, 
prior to the step of constructing the maximal rectangles, the image is filtered by 
detecting foreground separator elements that are objects in the foreground of the 
image having a pattern of pixel values deviating from said background property, and by 
replacing pixels of the detected foreground separators by pixels having the background 
property. 

However, Huang discloses a filtering step wherein the image is filtered by 
detecting foreground separator elements that are objects in the foreground of the 
image having a pattern of pixel values deviating from said background property 
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(column 14, lines 21-23, i.e. long line segments), and by replacing pixels of the 
detected foreground separators by pixels having the background property (see Figs. 
2C and 2D). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Huang's filtering method prior to the step 
of constructing the maximal rectangles of Belaid combined with Antonacopoulos, the 
motivation being to improve field extraction by removing extraneous lines (Huang: 
column 7, lines 49-53). 

Regarding claim 9, Belaid in combination with Antonacopoulos and further in 
view of Huang, teaches the method as claimed in claim 8, wherein the foreground 
separator elements include black lines (Huang: column 14, line 22), dashed lines, or 
dotted lines (Huang: column 19, BLOB, i.e. dashed and dotted lines are 
encompassed by the BLOB primitive). 

9. Claims 12, 13, 16 and 21 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Belaid as applied to claim 1 above, and further in view of Sakai et al. 
(Hereinafter "Sakai": US 5949555). 

Regarding claim 12, Belaid teaches the method as claimed in claim 1, wherein 
the step of constructing the separating elements comprises: detecting graphical 
elements that are objects in the foreground of the image having a pattern of pixel 
values deviating from said background property (page 744 § 2, i.e. lines). 
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Belaid does not teach the limitation wherein the separating elements are 
constructed around the graphical elements. 

However, Sakai discloses a method wherein the separating elements are 
constructed around the graphical elements (Figs. 30-32). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Sakai's separation method to Belaid's 
application, the motivation being to improve field extraction by precise classification of 
field attributes (Sakai: column 1, lines 47 and 48). 

Regarding claims 13, 16 and 21, Belaid teaches the method, equivalent 
device, and equivalent computer program product as claimed in claim 1. 

Belaid does not teach the limitation wherein at least one of the fields is classified 
as text field, a reading order is detected in the text field, and foreground components 
are joined to text lines in the text field in a direction corresponding to said reading 
order. 

However, Sakai discloses a method, equivalent computer program product, and 
equivalent device wherein at least one of the fields is classified as text field (column 
20, lines 17-19), a reading order is detected in the text field (column 20, lines 29- 
36, i.e. vertical or horizontal direction), and foreground components are joined to 
text lines in the text field in a direction corresponding to said reading order (e.g. see 
Figs. 30A, Title Portion). 
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Therefore it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to have applied Sakai's separation method to Belaid's 
application, the motivation being to improve field extraction by precise classification of 
field attributes (Sakai: column 1, lines 47 and 48). 

10. Claim 19 is rejected under 35 U.S.C. 103(a) as being unpatentable over as 
applied to claim 1 above, and further in view of Mahoney et al. (Hereinafter "Mahoney": 
US 6470095). 

Belaid teaches the device as claimed in claim 18. 

Belaid does not disclose the device further comprising: a display unit for 
displaying fields of the image after segmenting. 

However Mahoney discloses a display unit for displaying fields of the image after 
segmenting (Fig. 1, Display Monitor, and Fig. 21). 

Therefore it would have been obvious to one of ordinary skill in the art at the 
• time the invention was made to have applied utilized Mahoney's display device, the 
motivation being to provide user feedback (Mahoney: column 4, lines 1-3). 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Clifton G. Daley whose telephone number is 571-270- 
3144. The examiner can normally be reached on Monday - Friday 7:30am - 4:00pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Samir Ahmed can be reached on 571-272-7413. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO 
Customer Service Representative or access to the automated information system, call 
800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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